Accurate, Non-integer Bit Estimation for H.265/HEVC and H.264/AVC Rate-Distortion Optimization

Lagrangian Rate Distortion Optimization (RDO) is widely employed in modern video encoders such as H.264/AVC and H.264/AVC for mode decision. During the RDO process the distortion and bit rate of each mode has to be estimated and more an accurate estimation leads to better optimized decisions. This paper explores accurate, non-integer bit estimation and its impact on better RDO. Simulations show 0.2% to 13.0% bit rate saving for H.265/HEVC and 0.2% to 8.0% for H.264/AVC. Keywords—Non-integer/Fractional Bit Estimation; CABAC; Accurate Rate-Distortion Optimization;